Heavy equipment simulation system and methods of operating same

ABSTRACT

A heavy equipment simulation system for simulating an operation of a heavy equipment vehicle in a virtual environment is described herein. The heavy equipment simulation system includes a support frame, an operator input control assembly coupled to the support frame for receiving input from a user, a motion actuation system coupled to the support frame for adjusting an orientation of the support frame with respect to a ground surface, a display device assembly configure to display the virtual environment including the simulated heavy equipment vehicle, a virtual reality (VR) headset unit adapted to be worn by the user, and a control system.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/677,345, filed on May 29, 2018, all of which ishereby incorporated by reference in its entirety for all purposes.

COPYRIGHT NOTICE

The figures included herein contain material that is subject tocopyright protection. The copyright owner has no objection to thefacsimile reproduction by anyone of this patent document as it appearsin the U.S. Patent and Trademark Office, patent file or records, butreserves all copyrights whatsoever in the subject matter presentedherein.

TECHNICAL FIELD

The present invention relates to heavy equipment computer simulationsystems and methods of operating a heavy equipment computer simulationsystem.

BACKGROUND OF THE INVENTION

Machines such as excavators, wheel loaders, track-type tractors, motorgraders, large vehicles, and other types of heavy equipment are used toperform a variety of tasks associated with industries such as mining,farming, construction and transportation, for example. Operators ofthese machines may be required to receive a significant amount oftraining prior to operating these machines on-site. In some cases,machine operators must be licensed and certified by a certificationboard or governing body to operate certain equipment or vehicles toensure that the operator has received the appropriate training.

Machine operators are generally trained in computer-based simulators andin training exercises prior to performing actual work-relatedoperations. While these methods may provide a basic level of operationalexposure, they may not provide an environment that completely preparesthe operator for actual “real-world” work experiences associated with ajob site. Thus, many inexperienced machine operators may requireadditional on-the-job training in certain areas associated with machineoperation using costly fuel and manpower. Further, many experiencedmachine operators may require supplemental training for certainoperational skills and/or new techniques associated with one or moremachines. Such tasks add to the expense and time associated withcompleting business objectives.

Current simulation systems for heavy equipment are simplistic indisplaying the simulated environment as well as in calculating anddisplaying how the simulated machine interacts with the environment. Inaddition, teaching and instructor interaction is limited.

The present invention is aimed at one or more of the problems identifiedabove.

SUMMARY OF THE INVENTION

In different embodiments of the present invention, systems, methods, andcomputer-readable storage media for training an operator of a machineincluding interacting with a simulation mode of a heavy equipment modelsimulator and evaluating an operator using criteria substantiallysimilar to an original equipment manufacturer (OEM) criteria for anon-simulated heavy equipment model are described herein. Interactingincludes performing one or more simulated exercises substantiallysimilar to an OEM exercise for the non-simulated heavy equipment model.

In one embodiment of the present invention, a heavy equipment simulationsystem for simulating an operation of a heavy equipment vehicle in avirtual environment is provided. The heavy equipment simulation systemincludes a support frame, an operator input control assembly coupled tothe support frame for receiving input from a user, a motion actuationsystem coupled to the support frame for adjusting an orientation of thesupport frame with respect to a ground surface, a display deviceassembly configured to display the virtual environment including thesimulated heavy equipment vehicle, a virtual reality (VR) headset unitadapted to be worn by the user, and a control system. The operator inputcontrol assembly includes a heavy equipment vehicle control assembly foruse in operating a simulated heavy equipment vehicle in the virtualenvironment. The heavy equipment vehicle control assembly is configuredto generate first input signals based on the user's interaction with theheavy equipment vehicle control assembly. The VR headset unit includes aVR head mounted display and a position sensor. The VR head mounteddisplay is configured to display the virtual environment including thesimulated heavy equipment vehicle. The position sensor is configured todetect an orientation of the VR headset unit and generate second inputsignals based on the detected orientation of the VR headset unit. Thecontrol system includes a processor coupled to a memory device. Theprocessor is programmed to generate the virtual environment includingthe simulated heavy equipment vehicle, operate the simulated heavyequipment vehicle within the virtual environment based on the firstinput signals received from the heavy equipment vehicle controlassembly, operate the motion actuation system to adjust the orientationof the support frame with respect to the ground surface based onmovements of the simulated heavy equipment vehicle within the virtualenvironment, render the virtual environment including the simulatedheavy equipment vehicle on the VR head mounted display, and render thevirtual environment including the simulated heavy equipment vehicle onthe display device assembly.

In another embodiment of the present invention, a method of operating aheavy equipment simulation system is provided. The heavy equipmentsimulation system includes a support frame, an operator input controlassembly coupled to the support frame for receiving input from a user, amotion actuation system coupled to the support frame for adjusting anorientation of the support frame with respect to a ground surface, adisplay device assembly configured to display the virtual environmentincluding the simulated heavy equipment vehicle, a virtual reality (VR)headset unit adapted to be worn by the user, and a control system. Themethod includes the control system performing the steps of generatingthe virtual environment including the simulated heavy equipment vehicle,operating the simulated heavy equipment vehicle within the virtualenvironment based on the first input signals received from the heavyequipment vehicle control assembly, operating the motion actuationsystem to adjust the orientation of the support frame with respect tothe ground surface based on movements of the simulated heavy equipmentvehicle within the virtual environment, rendering the virtualenvironment including the simulated heavy equipment vehicle on the VRhead mounted display, and rendering the virtual environment includingthe simulated heavy equipment vehicle on the display device assembly.

In yet another embodiment, one or more non-transitory computer-readablestorage media, having computer-executable instructions embodied thereonare provided. When executed by at least one processor, thecomputer-executable instructions cause the processor to generate avirtual environment including the simulated heavy equipment vehicle,operate the simulated heavy equipment vehicle within the virtualenvironment based on the first input signals received from a heavyequipment vehicle control assembly, operate a motion actuation system toadjust the orientation of a support frame with respect to a groundsurface based on movements of the simulated heavy equipment vehiclewithin the virtual environment, render the virtual environment includingthe simulated heavy equipment vehicle on a VR head mounted display, andrender the virtual environment including the simulated heavy equipmentvehicle on a display device assembly.

In other aspects of the present invention, the methods of operating aheavy equipment simulation system may also include a method ofsimulating a physical interaction in a simulator. The method includesidentifying simulated physical contact zones between two or moresimulated objects and defining one or more interactions in the contactzones separate from simulated environmental or equipment object rules.Some embodiments relate to a machine-readable medium comprisinginstructions, which when implemented by one or more processors performthe following operations: initiating an instruction module, initiating asimulation module for controlling a machine, selecting an instructionmode, interacting with a simulation mode, compiling results,transferring results to instruction module and outputting results.

BRIEF DESCRIPTION OF DRAWINGS

Non-limiting and non-exhaustive embodiments of the present invention aredescribed with reference to the following figures. Other advantages ofthe present disclosure will be readily appreciated, as the same becomesbetter understood by reference to the following detailed descriptionwhen considered in connection with the accompanying drawings wherein:

FIG. 1 is a perspective view of a heavy equipment simulation system,according to embodiments of the present invention;

FIG. 2 is a schematic block diagram of a heavy equipment simulationsystem shown FIG. 1;

FIG. 3 is a schematic block diagram of a control system that may be usedwith the heavy equipment simulation system shown FIG. 1;

FIG. 4 is a perspective view of a portion of the heavy equipmentsimulation system shown in FIG. 1;

FIGS. 5-27 are illustrations of exemplary screenshots of graphical userinterfaces that may be generated and displayed by the control system foruse with the heavy equipment simulation system shown in FIG. 1,according to embodiments of the present invention;

FIG. 28 is another schematic block diagram of a control system that maybe used with the heavy equipment simulation system shown FIG. 1; and

FIGS. 29-33 are flowcharts illustrating various algorithms that may beexecuted by the control system for operating the heavy equipmentsimulation system, according to embodiments of the present invention.

Corresponding reference characters indicate corresponding componentsthroughout the several views of the drawings.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention provides a heavy equipmentsimulation system 10 for simulating an operation of a heavy equipmentvehicle in a virtual environment. The heavy equipment simulation system10 includes hardware and software for generating an enhanced heavyequipment simulation experience for a user in a way that more closelymerges the simulation with actual work operations. Additionally, theembodiments of the present invention describe unique instructor control,instruction modules and student or user feedback systems.

The invention relates generally to heavy equipment simulation and, moreparticularly, to a heavy equipment simulation system 10 that can swapbetween rendering to a monitor and a virtual reality (VR) deviceinstantaneously based on the VR HMD (head mounted display)position/state.

In some cases, users can encounter issues (motion sickness, discomfort,etc. . . . ) while using a HMD and may want to continue with the currentsimulated task while visualizing the virtual simulation through astandard monitor. The heavy equipment simulation system 10 includesheavy equipment training software that has two rendering modes. The tworendering modes include rendering to a monitor, and rendering to a VRHMD. The system 10 also allows the trainee user to switch between VR andstandard play modes, during runtime, without interruption. The system 10includes a mix of standard VR functionality combined with custom code toswitch back and forth between the standard in-game camera andfunctionality to a VR enabled camera with standard VR head trackingfunctionality.

In addition, the VR HMD has the ability to track the user's headmovement and position in 3D space, and this also includes a way ofdetermining if the user is actually wearing the HMD or not. When a userputs on or takes off a VR HMD, background code keeps track of thisstate. The system 10 is programmed to check these states and makechanges to how the program will try and render out the virtualenvironment scene. If the system 10 determines that a user is wearingthe HMD, standard camera views and camera switching methods aredisabled, and a VR ready camera is activated. If for any reason, at anytime, the user decides to take off the HMD, the standard camera andfunctionality is restored, and the VR enabled camera along with itsfunctionality is deactivated. This process can be repeated moving to andfrom the VR presentation at any time.

The desired functionality is all dependent on having everything withinthe game engine set to enable VR. Once this functionality is enabled, itis possible to get all the information needed about the attached HMD andstart controlling the system's output to the device. Both VR andstandard rendering methods need to be in place to: 1.) determine whatkinds of output the system can provide, and 2.) provide thefunctionality to switch between them.

Referring to FIGS. 1-3, in the illustrated embodiment, the heavyequipment simulation system 10 includes a support frame 12, an operatorinput control assembly 14, a motion actuation system 16, a displaydevice assembly 18, a virtual reality (VR) headset unit 20, and acontrol system 22 for operating the components of the heavy equipmentsimulation system 10. The heavy equipment simulation system 10 generatesand displays a virtual environment 24 including a simulated heavyequipment vehicle 26 (shown in FIGS. 1, and 4-6) and operates thesimulated heavy equipment vehicle 26 based on user commands received viathe operator input control assembly 14 to simulate the operation of aphysical heavy equipment vehicle.

The operator input control assembly 14 includes an operator seat 28 forsupporting the user from the support frame 12, a heavy equipment vehiclecontrol assembly 30, and a user input device 32. The heavy equipmentvehicle control assembly 30 includes physical vehicle input controls 34for use in operating the simulated heavy equipment vehicle 26. Thephysical vehicle input controls 34 are similar to the input controls andcontrol panels found in corresponding physical heavy equipment vehicleto enable the user to experience a real-life operation of a heavyequipment vehicle in a simulated virtual environment. The heavyequipment vehicle control assembly 30 is also configured to generatefirst input signals based on the user's interaction with the heavyequipment vehicle control assembly 30 and transmit the first inputsignals to the control system 22 indicating a desired operation of theuser. The user input device 32 may include a mouse and keyboard inputdevice that is configured to receive input from the user to enable theuser to operate the system 10.

The motion actuation system 16 includes a plurality of actuators 36 thatare coupled to the support frame 12 for adjusting an orientation of thesupport frame 12 with respect to a ground surface. The motion actuationsystem 16 may be operated to adjust the orientation of the support frame12 with respect to the ground surface based on movements of thesimulated heavy equipment vehicle 26 within the virtual environment 24.

The display device assembly 18 includes a plurality of display monitors38 that are coupled to a support assembly 40. The support assembly 40 isconfigured to support the plurality of display monitors 38 from theground surface. The display device assembly 18 is operated to displaythe virtual environment 24 including the simulated heavy equipmentvehicle 26 to the user. The display device assembly 18 is spaced adistance from the support frame 12 to enable the user to view thevirtual environment 24 being displayed on the plurality of displaymonitors 38 when the user is seated in the operator seat 28.

The VR headset unit 20 is also operated to display the virtualenvironment 24 including the simulated heavy equipment vehicle 26 to theuser. The VR headset unit 20 is adapted to be worn about the head of theuser and is configured to render the virtual environment 24 in a morerealistic virtual reality setting to the user. The VR headset unit 20includes a VR head mounted display 42 and one or more position sensors44 mounted to the VR headset unit 20. The VR head mounted display 42 isconfigure to display the virtual environment 24 including the simulatedheavy equipment vehicle 26 in a virtual reality setting. The positionsensors 44 are configured to detect an orientation of the VR headsetunit 20 and generate second input signals based on the detectedorientation of the VR headset unit 20. The VR headset unit 20 may alsoinclude a hand tracking component 46 that includes one or more infraredsensors mounted to the VR headset unit 20 and that are configured todetect a position of the user's hands.

Referring to FIGS. 1-3, in one embodiment, the heavy equipmentsimulation system 10 includes a hardware platform whose main componentsconsist of a tubed metal frame for use with the support frame 12 and ametal display stand for use with the display device assembly 18. Theframe 12 contains an operator seat 28, the heavy equipment vehiclecontrol assembly 30 including a plurality of machine control pods 34which are connected via RJ45 network connections to a Network AreaConcentrator (NAC) 47 that communicates via a proprietary CAN/J1939protocol to carry all of the control signals back to the control system22 over a Universal Serial Bus (USB) connection. The machine controlpods 34 are secured to the frame and contain machine specific OriginalEquipment Manufacturer (OEM) controls that serve as inputs to thesimulation software 48 executed by the control system 22. The motionactuation system 16 contains a 3 degree-of-freedom (DOF) motion systemthat is communicatively coupled to the control system 22 over a USBconnection. The motion actuation system 16 has two motion ActuatorControl Modules (ACM) 50, 52 mounted underneath the frame 12 which areconnected via shielded RJ45 network connections. Each ACM 50, 52 is thenconnected to two actuators. A first ACM 50 is connected to a first pairof actuators 36, and a second ACM 52 is connected to a second pair ofactuators 36. The user input device 32 contains a wireless mouse andkeyboard. The support assembly 40 includes a display stand containssupports the control system 22 which includes a computer processingdevice 54, which contains a processor 56, memory 58, storage 60, andgraphics card 62. A specific example of a suitable computer platform isa Dell5820 but it is to be understood that the teachings herein can bemodified for other presently known or future hardware platforms. Aspecific example of a suitable computer graphics card is a NVIDIAGTX1080 but it is to be understood that the teachings herein can bemodified for other presently known or future cards. The display stand 40contains an audio speaker 64 that is connected to the computerprocessing device 54 over a USB connection. The support frame 12 has twopositional tracking sensors 66 that are communicatively coupled to thecomputer processing device 54 over USB connections. The display stand 40has three display monitors 38, which are connected to the computerprocessing device 54 graphics card 62 over HDMI and/or DisplayPort forrendering a simulated view. The display monitors 38 may include LEDdisplays, LCD displays, or any suitable display device for displayingcomputer-generate images. The heavy equipment simulation system 10currently has three displays 38 acting as a single view, but it is to beunderstood that the teachings herein can be modified for other presentlyknown or future combinations of displays acting as multiple views. Theheavy equipment simulation system 10 also includes the VR headset unit20 that is connected to the computer processing device 54 over USB andto the graphics card 62 over HDMI. A specific example of a suitable VRheadset unit 20 is the Oculus Rift™ but it is to be understood that theteachings herein can be modified for other presently known or futureHead Mounted Displays. The VR headset unit 20 also interacts with thepositional tracking sensors 66 over infrared using outside-in trackingto send specific locations of the HMD to the software 48. The heavyequipment simulation system 10 contains a hand tracking component 46which is mounted on the VR headset unit 20 but communicates directlywith the computer processing device 54 over USB.

While the machine-readable medium 60 is shown in an example embodimentto be a single medium, the term “machine-readable medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database, and/or associated caches and servers) thatstores the one or more sets of instructions. The term “machine-readablemedium” shall also be taken to include any medium that is capable ofstoring, encoding or carrying a set of instructions for execution by themachine and that cause the machine to perform any one or more of themethodologies of the embodiments of the present invention. The term“machine-readable medium” shall accordingly be taken to include, but notbe limited to, solid-state memories, optical and magnetic media, andcarrier wave signals.

Referring to FIG. 3, in some embodiments, the control system 22 mayinclude one or more software modules 68 that include computer-executableinstructions that are executed by the processor 56 to perform thealgorithms illustrated in FIGS. 29-33 and described herein, to operatethe heavy equipment simulation system 10. A memory device 70 includingthe memory 58 and storage 60 may also store one or more data files 72that include information that may be used by the processor 56 whenexecuting the computer-executable instructions for each software modules68. The memory device 70 includes at least some form of computerreadable media. By way of example and not limitation, computer readablemedia may include computer storage media including volatile andnonvolatile, removable and non-removable media implemented in any methodor technology that enables storage of information, such as computerreadable instructions, data structures, program modules, or other data.

In the illustrated embodiment, the control system 22 includes a virtualsimulation module 74, a VR camera module 76, a non-VR camera module 78,a display rendering module 80, a control input module 82, a motioncontrol module 84, a performance tracking module 86, and a performancemeasurement module 88.

The virtual simulation module 74 generates the virtual environment 24including the simulated heavy equipment vehicle 26 using the objectrender files and vehicle attribute files stored in the memory device 70.The VR camera module 76 is programmed to establish a VR camera viewpointlocation within the virtual environment 24 that is used to render a VRcamera viewpoint based on the VR camera viewpoint location. For example,in one embodiment, the VR camera module 76 establishes the VR cameraviewpoint location within a cab of the simulated heavy equipment vehicle26 to provide the user with a first-person view sitting with the cab ofthe simulated heavy equipment vehicle 26. The VR camera viewpoint isused to render images of the virtual environment on the on the VR headmounted display 42. In some embodiments, the virtual simulation module74 may include a Unity™ game engine.

The non-VR camera module 78 is programmed to establish a non-VR cameraviewpoint location within the cab of the simulated heavy equipmentvehicle 26 within the virtual environment 24 that is used to render anon-VR camera viewpoint of the virtual environment 24 on the displaydevice assembly 18 based on the non-VR camera viewpoint. In oneembodiment, the non-VR camera viewpoint location is adjacent to the VRcamera viewpoint location within the cab of the simulated heavyequipment vehicle 26.

The display rendering module 80 is programmed to render images of thevirtual environment 24 on the display device assembly 18 and the VR headmounted display 42. The display rendering module 80 is also programmedto operate in a VR display mode to simultaneously render and display theVR camera viewpoint on the VR head mounted display 42 and the displaydevice assembly 18. The display rendering module 80 is also programmedto operate in a non-VR display mode to render and display the non-VRcamera viewpoint of the virtual environment 24 on the display deviceassembly 18.

The control input module 82 is programmed to receive first input signalsindicating the user's interaction with the heavy equipment vehiclecontrol assembly 30 and operate the simulated heavy equipment vehicle 26within the virtual environment 24 based on the first input signalsreceived from the heavy equipment vehicle control assembly 30.

The motion control module 84 is programmed to operate the motionactuation system 16 to adjust the orientation of the support frame 12with respect to the ground surface based on movements of the simulatedheavy equipment vehicle 26 within the virtual environment 24.

The performance tracking module 86 is programmed to implement asimulated training exercises within the virtual environment thatrequires the user to operate the simulated heavy equipment vehicle 26through a series of predefined tasks. The performance tracking module 86also monitors various performance parameters associated with the user'soperation of the simulated heavy equipment vehicle 26 during thesimulated training exercises.

The performance measurement module 88 is programmed to receive themonitored performance parameters from the performance tracking module 86and generate various evaluation scores based on the performanceparameters and output various performance reports to display theevaluation scores. In some embodiments, the performance tracking module86 and/or performance measurement module 88 include SimU Campus™Training Management software.

Motion/Actuators: The motion system 16 is being sent data from thevehicle's rigid body chassis setup. This data consists of the chassis'yaw, pitch, roll, angular velocity, and linear velocity. Along with thechassis telemetry data there is also a need for information from thetracks or tires which vary depending on the machine. From a trackedmachine all that is needed is the ground speed of the track, but forwheeled vehicles there is a need to know information from the outer mostfour tires. The tire information needed includes tire load, tire slipand suspension travel. Along with this data there is also a need for theangular acceleration of the front most and rear most axles.

Once gathered from the vehicle, all this data is transmitted into theprovided D-BOX motion DLL. This DLL is created and provided by D-BOX forour use. The DLL consumes the information and translates that into thelinear actuators connected to the PC via USB. The actuators are boltedto the simulator frame and are each connected to one central hub that isreceiving the signals from the PC's USB output. They are capable oflifting the simulator frame vertically with each actuator movingindependently to provide pitch (front/rear tilt), roll (left/righttilt), and heave (vertical up/down motion).

The motion system receives the vehicle's chassis orientation (yaw,pitch, roll), local velocity, and angular velocity. From the engine, themotion system receives engine power, max rpm, red line rpm, idle speed,engine rpm, and transmission gear. Some variables that are sent in fromthe vehicle can depend on if the vehicle is a tracked or wheeledvehicle. If the vehicle has tracks, the left and right track speeds aresent, but if the vehicle has wheels each tire's wheel slip, load, andacceleration are sent. The final items that the motion system receivesare suspension travel from the front and rear right and left suspensioncomponents. The motion DLL takes the values and converts those into thesignals being sent to the motion actuators. This is all handled by theDBox motion software.

Camera Frame: The cameras within the simulator are elements very similarto the objects used to make up the different parts of the vehicle modelitself, except they are standard components of the Unity™ game engineand require no extra coding to render out the 3D images. Unity's camerasare used in a few different ways to obtain the desired results for thesimulators, but all consist of taking a camera object and attaching itto a base object that provides extra functionality to move the cameraaround the 3D environment.

These base objects are used to give the camera the ability to freelymove around in 3D space, orbit around the vehicle (rotate 360 degreesaround the vertical axis of the chassis and close to 180 degrees aroundthe horizontal axis while focusing on the chassis), act as the user'smain in-cab view (simulating the users head and allowing limitedrotation), and attaching the virtual reality (VR) camera to themachine's chassis. When the camera is attached to a mount that isattached to the chassis (in-cab mount/VR mount) the camera becomes partof the chassis and will follow the vehicle's position as it moves in 3Dspace.

The in-cab mount will allow the user to rotate the camera a certainamount vertically and horizontally (amount can vary depending on themachine) using the mouse. The VR mount is similar to the in-cab mountbut all camera position and rotational movement is being input by theuser's head mounted display (HMD) and the spatial tracking camerasprovided with the VR device. The orbit mount is similar to the in-caband VR mounts in the fact that they are all attached to one position onthe machine's chassis. This means that no matter the position orrotation around the vehicle, the orbit camera will follow the machinethroughout the environment. The free fly mount is usually not allowed inthe production version of the simulator, but can be provided in custombuilds for customers. This camera mount is special in that it is theonly mount that is not attached to the vehicle chassis. Using mouse andkeyboard input, the free fly mount can move the camera anywhere in the3D environment with the keyboard providing positional movement and themouse controlling the rotational movement.

Oculus Rendering. As discussed in the Camera Frame portion, thesimulator uses a mount system to attach cameras to the machine'schassis, but the camera itself contains all the game engine specificcode that converts the standard camera view to a VR presentation. Thisrequires a few items to be set within Unity before any of the VRfunctionality will work. When using an Oculus™ HMD 20, Unity™ will needthe Oculus asset packages to be installed, Unity's XR settings have tobe enabled, and the Oculus SDK has to be added to the list of VirtualReality SDKs. When all of these settings are correct the simulator canbe rendered into the HMD.

With this base functionality enabled the foundation of the unique VRswitching can be setup. Each camera that is intended to render to ascreen and not in VR will need to be set to render to the desiredscreen. Once the standard cameras are setup, the VR camera can now runindependently.

The VRDeviceConnectionManager.cs script controls the majority of theinformational displays (VR ready/not ready, remove headset, andcalibration instructions displays), but also controls what cameras areactive to reduce rendering overhead. Cameras can be added to this scriptto enable and disable cameras depending on if the user is wearing theheadset or not. Beyond switching cameras; one of this scripts mainduties is to enable/disable VR depending on certain conditions. Sinceone of the main goals of the simulator is to give operators the feelingof the realistic controls and their placement; VR is not enabled untilafter the machine has been started and is ready to move. This allowsoperators to gain muscle memory and spatial awareness of what controlsare needed to begin machine operation. When the machine is ready to beshutdown VR will be disabled and the user will remove the headset anduse the hardware controls to perform the shutdown procedures. Anotherdeciding factor in VR being enabled or disabled is the Leap Motioncontroller. If the Leap Motion controller is not working properly or notdetected VR will be disabled and the VR Not Ready badge will bepresented on screen. This is also true of the VR headset itself; if itis not detected or is malfunctioning the VR Not Ready badge will bedisplayed. If all devices are present and working, and startupprocedures have been passed, the VR Ready badge will be displayed andthe user is free to put on the headset. If the user removes the headsetat any time during the simulation the VR camera will be disabled and thestandard cameras will be enabled, and the simulation will continue, butthe rendering will switch from the VR headset to the standard screens.

All the 3-D models remain the same for VR and non VR use, but there is adedicated VR camera within the virtual environment. Along with the VRcamera there is a non VR camera that resides in the same position as theVR camera. Both have separate methods of control as the VR camera isreceiving signals from the headset as to where it should be positionedand how it should be oriented, and the non VR camera's orientation canbe controlled using the mouse. The TV monitors show the view from one ofthe “eyes” of the VR headset when the headset is in use. When it is notin use the TVs display the simulator from the view of the non VR camera.Within Unity (the game engine), code is running that checks a sensorattached to the headset. This sensor checks to see if there is anythingclose to the inside of the headset. Once it senses something the gameengine sets a flag that says the headset is in use and position andorientation data is now available. Once this flag is set in the code,the control system 22 created checks to see what cameras are active inthe scene and deactivates cameras that are not presenting in VR andactivates the VR camera. There is only one VR camera in the scene.

In the illustrated embodiment, the processor 56 of the control system 22is programmed to execute a plurality of algorithm steps to operate theheavy equipment simulation system 10 to simulate a training exercise.For example, to execute the training exercise, the processor 56 may beprogrammed to execute the algorithm steps including generating thevirtual environment 24 including the simulated heavy equipment vehicle26, and operating the simulated heavy equipment vehicle 26 within thevirtual environment 24 based on the first input signals received fromthe heavy equipment vehicle control assembly 30. The processor 56 alsooperates the motion actuation system 16 to adjust the orientation of thesupport frame 12 with respect to the ground surface based on movementsof the simulated heavy equipment vehicle 26 within the virtualenvironment 24.

The processor 56 also renders the virtual environment 24 including thesimulated heavy equipment vehicle 26 on the VR head mounted display 42,and renders the virtual environment 24 including the simulated heavyequipment vehicle 26 on the display device assembly 18.

In the illustrated embodiment, the processor 56 is programmed to operatethe heavy equipment simulation system 10 in the VR display mode byestablishing the VR camera viewpoint location within a cab of thesimulated heavy equipment vehicle 26 within the virtual environment 24,rendering a VR camera viewpoint of the virtual environment 24 on the VRhead mounted display 42 based on the VR camera viewpoint location. Inaddition, when operating in the VR mode, the processor 56 monitors theorientation of the VR headset unit 20 based on the second input signalsreceived from the VR headset unit 20, and adjusts an orientation of therendered VR camera viewpoint based on changes in the orientation of theVR headset unit 20.

The processor 56 may also be programmed to operate in the VR displaymode by simultaneously rendering the VR camera viewpoint of the virtualenvironment 24 on the display device assembly 18 and the VR headset unit20.

In one embodiment, the processor 56 may be programmed to receive signalsfrom the hand tracking component 46 mounted to the VR headset unit 20and operate in the VR display mode by rendering images of simulated userhands 90 (shown in FIG. 4) within the virtual environment 24 based onthe detected position of the user's hands.

The processor 56 is also programmed to operate the heavy equipmentsimulation system 10 in the non-VR display mode. In the non-VR displaymode, the processor 56 establishes a non-VR camera viewpoint locationwithin the cab of the simulated heavy equipment vehicle 26 within thevirtual environment 24 and renders a non-VR camera viewpoint of thevirtual environment 24 on the display device assembly 18 based on thenon-VR camera viewpoint location. In addition, the processor 56 isprogrammed to not render the VR camera viewpoint on the VR head mounteddisplay 42 when operating in the non-VR camera mode. For example, uponenabling the non-VR camera viewpoint of the virtual environment, theprocessor 56 may disable the VR camera viewpoint.

When operating in the non-VR mode, the processor 56 is also programmedto receive third input signals from the user input device 32 based onbased on the user's interaction with the user input device 32. The userinput device 32 is configured to generate the third input signals basedon the user's interaction with the user input device 32 for adjusting anorientation of a displayed viewpoint of the virtual environment 24. Theprocessor 56 is programmed to operate in the non-VR display mode byadjusting an orientation of the rendered non-VR camera viewpoint basedon the third input signals received from the user input device 32.

The processor 56 is also programmed to operate in the VR mode or thenon-VR mode based on the detected orientation of the VR headset unit 20.For example, the VR headset unit 20 is adapted to be worn on a head ofthe user. The processor 56 may be programmed to switch from operating inthe VR mode to operating the non-VR mode upon detecting the VR headsetunit 20 has been removed from the head of the user.

Vehicle Documentation:

The following describes the algorithms and concepts used by the controlsystem 22 in vehicle simulations and the process of building a basicsimulated heavy equipment vehicle 26 within the virtual environment 24.

Powertrain. The powertrain consists of several parts (such as engines,transmissions, differentials, etc.) linked together with shafts. In thepowertrain simulation model, torque and inertia is always passed fromthe torque generator, downstream (e.g. Engine->Transmission->Wheels) andthe angular speed is solved and always passed upstream(Wheels->Transmission->Engine).

Shafts. Shafts exist to link parts together and relay data from one partto the next. This data includes Torque, Speed, Inertia, and whether thetorque is a braking torque.

Parts. Parts will generally define several input and/or output shaftsthat allow the control system 22 to link them together. The parts areresponsible for modifying powertrain data and the shafts are responsiblefor relaying the modified data. Power generation and speed solving canhappen within many different parts. For example, a torque converter willsolve speed for its input shaft and generate torque for its outputshaft, based on the speed difference between the two. The control system22 can typically create a vehicle using the premade parts that have beendeveloped, and users can always create custom part by inheriting fromPartScript and implementing a unique Step( )function.

Premade Parts: the control system 22 includes a list of parts built intothe shared library: AxleScript, CurveBasedTorqueConverter, DifferentialScript, EngineScript, SteeringDifferentialScript, TorqueConverterScript,TransmissionScript.

Wheels: A custom wheel collider has been developed to better supporttire simulations. The wheel uses a typical spring and damper type ofsimulation for the suspension forces, but the control system 22 alsoapplied spring and damper concepts for tire deflection. The built-inwheel collider that ships with Unity™ only applies friction force to awheel to move it towards a zero velocity. The equation is something likethis: Force=−Velocity*Friction.

However, this means that no force gets applied if velocity is 0. Otherforces may act upon the machine causing it to move a little bit duringthe next simulation step. The friction forces would not get applieduntil the frame after the wheel starts moving. The result is a cycle ofmovement and friction forces, occurring alternately, allowing themachine to slowly slide along the ground over time. This is especiallyproblematic when high outside forces are being applied to the machine,such as when parking on a hill.

Tire Deflection Forces: Our wheel addresses sliding issues by using aspring and damper system to represent the flexing rubber forces on atire (tire deflection). Our wheels maintain a contact point on theground that is advanced as a tire spins or slips. When a machine isparked on a hill, spring and damper forces move the wheels towards theircontact point. There is a maximum amount of force that the spring canapply to push towards the contact point, and if that is exceed, thecontact point can move, simulating the breaking of traction, and themachine begins to slide. By using this model, the control system 22eliminates the Velocity portion of the equation when static friction isin effect, and a machine with 0 velocity can have forces applied tocounteract outside pushes and pulls by simulating the spring movementallowed with tire deflection. The forces are based on distance from thecontact point rather than the velocity of movement. When the tire beginsslipping, the control system 22 uses a combination of the tiredeflection forces as well as a velocity modifier to apply forces back tothe machine. Everything about how the machine is propelled goes throughthis tire deflection logic. To move the machine forward, the Axlesimulates a new wheel speed by reading torque data from upstream parts,and accumulating wheel forces to generate an acceleration value. Thiswheel speed is used to advance the contact point forward, causing thedeflection spring to apply a force to the chassis to propel the vehicleforward. All outside forces acting on a machine affect the powertrainthrough tire deflection. For example, digging resistance forces appliedto the machine keep the machine chassis held back while tire rotationmoves the contact point forward. The deflection spring generates forcedue to this separation in the wheel position and its contact point, andthat force is used to compute a torque on the wheel which slows thewheel down. This reduction in speed is propagated through the powertrainand may cause the engine bog down.

Visualizing Powertrain Design: FIG. 7 illustrates an example powertrainconfiguration.

Powertrain in Unity™ software: When applying powertrain components inUnity™, the control system 22 will have a series of part and shaftcomponents to build out the simulation. FIG. 8 illustrates an exampleshows a bull dozer component layout.

Creating The Prefab:

The vehicle should be built using a prefab whose pivot is positioned atground level so a vehicle can easily be dragged into the scene. To dothis, the control system 22 might create a ground plane at (0,0,0), andthen create an empty game object at that same position. The controlsystem 22 may be used to rename the game object and then drag it fromthe “Hierarchy” window (as shown in FIG. 9) into the “Project” window tocreate the prefab.

Build the vehicle's physical presence:

For this step, the user will need to bring in the vehicle's models(physical and visual), add collider components, rigid bodies, and jointup any parts that need to be constrained. For the most part, the controlsystem 22 can use generic Unity constraints. The software may have somespecial constraints that specifically help with heavy equipmentdevelopment scenarios.

Special constraints: HydraulicScript—the control system 22 has ahydraulic class that supports the specific functions that the controlsystem 22 need to simulate, such as overpowering a pressure reliefvalve, float mode, and velocity targets.VelocityRevoluteConfigurableScript—The velocity revolute script isuseful for simulating a velocity based hydraulic motor, such as theswing drive on an excavator.

Finish Assembly: the control system 22 should now have something thatrepresents a vehicle with colliders, models, rigid bodies, and joints.The rigid bodies have a somewhat realistic mass as a whole for thevehicle, with each object's mass within a 1:10 ratio of the otherobjects. Apply this to the prefab (as shown in FIG. 10).

Building the Powertrain:

FIGS. 10-16 illustrate various display screens that may be displayed bythe control system 22 to assist the user in building a vehiclepowertrain for use with simulated heavy equipment vehicle 26. In theillustrated example, the control system 22 enables a user to set up atank that uses differential steering. Powertrain GameObject—Create anempty game object as a child of the prefab named powertrain. Assign aPowertrainSimulationScript component to this game object (shown in FIG.11). Adding Powertrain Components and Connecting Shafts—Add additionalcomponents to the powertrain game object to design the powertrain. Thecontrol system 22 will use an engine, a curve-based torque converter, atransmission, and a steering differential in this case. The engine'storque curve, brake torque curve, and the transmissions gear ratios areconfigured to the specifications. Shaft components are placed in betweeneach part and assign the shaft to the output of the component above andthe input of the component below, as shown in FIG. 12). Configuring thePowertrain Simulation—Now the control system 22 needs to add each Partto the powertrain simulation component in the order the control system22 will need them to simulate, as shown in FIG. 13 (typically from thepower generator to the wheels). Add Some Wheels—The steeringdifferential has wheel assignment lists so that the speed calculationscan be applied to both tracks at the same time, as shown in FIG. 14.This allows for faster convergence on a good simulation result. If thecontrol system 22 has colliders on the tracks, remove them, or put theminto a collision layer that doesn't collide with the terrain. For atrack, the control system 22 will create multiple wheels inline to givemany contact points along the length of the track. For each wheel, add agame object as a child of the track and assign a WheelScript component.Assign the Chassis for which the control system 22 wants the wheels toapply forces. Connect Wheels to the Powertrain—as shown in FIG. 15, thecontrol system 22 allows the user to connect the wheels to thepowertrain. In many cases, the control system 22 would assign the wheelsto AxleScripts that have been created during the powertrain setup. Forthe tank, the control system 22 will assign the wheels to theSteeringDifferential (perform this for the left track and the righttrack). Adjust the wheels spring/damper—as shown in FIG. 16, the controlsystem 22 allows the user to run and adjust the spring and damper valuesfor the wheels until they properly support the vehicle.

The Scripts:

The control system 22 will need an input manager and a vehiclecontroller to drive the vehicle (at a minimum). In many cases, thecontrol system 22 will also have a transmission controller to run theautomatic shifting of the transmissions.

Input Manager:

The input manager is responsible for taking input from the hardware,sanitizing it into a logical format, and exposing it to the vehicle.There will be an input manager for each type of hardware the controlsystem 22 may use with the vehicle. The control system 22 will likelyhave a KeyboardInputManager and a HardwareInputManager, but any numberof additional input managers could be built. When sanitizing input, theuser should always choose the type of data that makes the most sensefrom the vehicle developer's perspective. For example, some hardwareuses 4 buttons that could be true or false to indicate the throttleamount. The input manager should convert the 4 boolean values into athrottle percentage that the vehicle developer can easily consume. Asadditional input managers are created, abstracting away the hardwarespecific details like this will make the life much easier compared totrying to emulate the 4 booleans with new hardware that now uses aslider control.

Vehicle Controller:

The vehicle controller is responsible for relaying input from the activeinput manager to the proper components of the vehicle. For example,relaying the throttle amount to the engine script, or applying a brakeforce to the wheels when the parking brake is on. TransmissionController—The transmission controller is responsible for handlingtransmission shifting, neutralization, gear selection, etc. This couldpossibly be made into a shared, generic class someday, offloading someof the vehicle specific logic to the vehicle controller, but at thispoint, it has been easiest to develop custom controllers on a perproduct basis. Apply the scripts to the vehicle—Assign the vehiclecontroller and input manager onto the root game object of the prefab.

Dynamic Terrain Documentation:

The following describes the algorithms and concepts used by the controlsystem 22 to use the dynamic terrain plugin and it also describes how tosetup various dynamic terrain objects within the virtual environment 24that may be influenced by the simulated heavy equipment vehicle 26.

Overview of Heightmap Based Terrain Concepts:

The terrains in our simulation use a grid of height data to create aheightmap geometry. This geometry is generated by placing vertices in apredefined grid on the X (right) and Z (forward) axes and using heightdata on the Y (up) axis to shape the height field. For efficiency, thecontrol system 22 uses cuboid volume calculations when determining howmuch material a height change represents (sample spacing X*samplespacing Z*height).

Digging Resistance:

Digging resistance is applied to the machine by creating joints that usedamping forces to drive towards a zero velocity. Damping forces aredetermined by multiplying the depth of the implement against apre-defined resistance curve. Low density material applies lessresistance force than high density material by using a predefinedfractional multiplier:Force=Curve.Evaluate(HighDensityDepth+LowDensityDepth*LowDensityModifier).

Resistance Easing:

Resistance easing is the process of looking ahead of the implement tocheck depths and reduce resistance based on how much lower the furtherout material is. This is useful for example, when pushing through a berm(A.) which should not apply as much resistance as pushing into the sideof a hill (B.), as shown in FIG. 17.

Erosion:

Erosion is handled by comparing two height samples against each other,in the direction of an applied acceleration, while consideringstart/target angles of repose that define when erosion should begin andto what angle the erosion should result in. The angles supplied are thevalues used under normal gravitational acceleration (9.81 m/s2),however, changes in the acceleration vector will affect the actualangles used. Erosion is handled efficiently by creating a maximumdeviation grid that is computed in the local frame of the heightmapbeing eroded. The control system 22 computes the height threshold thatthe 4 samples around a particular sample must be above or else erosionof the current sample will be triggered. These computed values definethe maximum deviation grid, and the control system 22 uses this grid tocompute once and compare many times against each sample in theheightmap.

Heightmap Supported Areas:

Heightmap supported areas are oriented bounding boxes that preventmaterial from eroding into them. When comparing a sample to thedeviation grid, if the adjacent sample is in a heightmap supported area,erosion will not be triggered in that direction. This is useful when itis desired to have the edge of a blade to prevent a berm from erodinginto the blade, for example.

Material Density Effects:

Erosion is currently applied only to low density materials. Theexclusion of high density materials allows operators to create steepedges when cutting through packed material.

System Components:

In some embodiments, the control system 22 includes the following systemcomponents:

1) TerrainManger:

The TerrainManager provides developers with singleton access to theterrain systems, generating filtered TerrainInfo objects based on thecriteria that the calling code sets. The control system 22 will need toadd a TerrainManager into the scene to make use of the dynamic terrainplugin.

2) DynamicTerrain:

The DynamicTerrain class defines a terrain object to be used by theterrain manager. The DynamicTerrain may include a grid of objects thatthe control system 22 will need to treat as a single terrain and willhandle stitching the changes across all objects. The DynamicTerrain isalso responsible for telling its children when to apply cached changes.The control system 22 caches changes rather than applying themimmediately because changing geometry is expensive, so it is desired todo it once per frame.

3) HeightsInfo Objects:

HeightsInfo exists to hide the complexity of working across multipleterrains by presenting the developer with a single heightmap that fitsthe requested working area (as shown in FIG. 18). The HeightsInfo objectis supported by a number of underlying classes that give access to theterrains that the HeightsInfo might affect. TerrainInfo objects. TheTerrainInfo object contains data for an entire, single heightmap. Thatheightmap could be a Unity terrain, a DirtHeightMap, a MoundHeightMap,or any heightmap that has a corresponding implementation of aTerrainInfo (See UnityTerrainInfo.cs, MoundHeightmapTerrainInfo.cs,DirtHeightmapTerrainInfo.cs). These specific TerrainInfo implementationsare responsible for caching and applying changes to the heightmap thatit represents. TerrainHeights Objects. A terrain heights object is ageneric subsection of the heightmap exposed through the TerrainInfoobjects. This subsection allows the developer to work with smallerarrays of height data that better fit the area that they are trying tomodify. When working across multiple terrains, the HeightsInfo objectthat the control system 22 retrieves from the TerrainManager may containmultiple TerrainHeights objects. However, the HeightsInfo managesapplying the changes to the correct TerrainHeights object.

4) Terrain Modifier Objects:

Terrain modifiers are used to modify the terrain and report terrainchanges to a listening component (for example, a BucketController thatmay want to add the modified volume into a DirtHeightMap inside of thebucket). Currently, the control system 22 supports 2 types of terrainmodifiers: TerrainModifierScript—this is a basic terrain modifier thatuses an oriented bounding box to determine which samples in a height mapare inside of the bounds, and it then lowers those samples to move themtowards the edge of the bounds; ColliderTerrainModifier—theColliderTerrainModifier uses raycasts to find where the sample liesrelative to its child colliders. The resulting contact point (assumingthe raycast hits a collider) is then used to lower the sample.

5) DepthSamplers:

Depth samplers are used to determine how deep into the ground a certainpoint is. Depths are returned as an array of depths representingdifferent layers of material and the depth in each. At the moment, only2 layers are supported; low density (loose) and high density (packed)material. The system is designed for future expansion into any number oflayers.

Creating an environment with a Unity terrain:

In some embodiments, the control system 22 is configured to allow usersto set up a virtual environment for use with a dynamic terrain. Thefollowing algorithm steps are executed by the control system 22:

Create an empty game object in the scene and name it TerrainManager. Adda TerrainManager component to the new object. If only using a singleterrain: create a Unity terrain and add a DynamicTerrain object to theterrain. If more than one terrain stitched together and behaving like asingle terrain: terrains must be the same resolution and aligned in anice checkerboard pattern; create an empty game object namedDynamicTerrain and add a DynamicTerrain component to it; assign all theterrain objects as children of the DynamicTerrain game object in thehierarchy.

Resize terrain to have an acceptable sample density. Add 2 textures tothe terrain (by default, the second texture is used for areas ofdigging, but can be changed on the TerrainManager component). Flattenthe terrain with some height. A default terrain as created cannot belowered. It must have height added using either brushes or the flattenfeature in order to allow for digging. Be sure to give enough height toallow sufficient digging for the application. If the terrain is loweredback down to its pivot, the samples cannot go any lower. Add a Box tothe scene and place a ColliderTerrainModifier component onto it. Pressplay and drag the box through the terrain in the editor. It should digout the terrain wherever it goes.

Setting a Bucket Up for Digging:

Once the control system 22 has the bucket geometry created and importedinto the simulation, the control system 22 can begin adding thecomponents required to make it dig. First, make sure the bucketcolliders are in a proper collision group (probably “Implement”). Add achild game object named “TerrainModifier” to the bucket. Add aTerrainModifierScript to this new object and adjust the oriented boundsto encompass the cutting edge and the opening of the bucket (see FIG.19).

Using the DirtHeightmap component:

The DirtHeightmap class is used to represent a miniature terrain thatban be positioned and rotated in any fashion in the environment.DirtHeightmaps are often used in conjunction with an ErodeAndSpawncomponent so the material can spawn TerrainEntities to transfermaterial. This type of terrain is useful for representing a pile of dirtin a bucket or in a truck bed (see FIG. 20). To create a DirtHeightmap,add a new, empty game object underneath the bucket in the hierarchynamed “DirtHeightmap”, as shown in FIG. 21.

Assign a DirtHeightmap component to the new game object and set the“Shrink Wrap Layer” to the same collision layer as the bucket. Thistells the DirtHeightmap to raycast and find where the bottom of thebucket is. It allows the mesh to hide when there is no material. Assigna dirt material to the MeshRenderer on the DirtHeightmap, as shown inFIG. 22.

Add a DynamicTerrain component to the DirtHeightmap game object. Asshown in FIG. 23, move the heightmap to the lower left corner of thebucket opening as pictured below and position it just above thecolliders that shrink wrap against.

Adjust the heightmap resolution, width, and length to match the bucket'sopening dimensions. When the control system 22 runs, with theDirtHeightmap selected, the control system 22 should be able to see thephysics collider of the heightmap to give a sense of how it fits to thebucket, as shown in FIG. 24.

Add the Heightmap into the TerrainModifiers “Terrains to Ignore” list.This prevents the bucket's terrain modifier from digging out theDirtHeightmap material, as shown in FIG. 25. Add a BucketController thathooks into the TerrainModifier's OnModify( ) callback to add material tothe DirtHeightmap.

Setting a Blade Up for Digging:

Create an empty child on the blade named TerrainModifier and assign aTerrainModifier component. Adjust the OrientedBoundingBox of theTerrainModifier to represent where to modify the Terrain in front of theblade. Add a MoundHeightmapController onto the blade to spawn andcontrol the mound.

Create a mound prefab to represent moving material:

Add the DirtRemeshing and DynamicTerrain components to the mound prefab.Create an empty child named Moundheightmap and add the MoundHeightmapcomponent. This game object will need to be offset on X by half of themound width as the mound will be drawn to the right of the pivot. Assigna material to the MoundHeightmap MeshRenderer. Add an empty child to theprefab named BladeEdgeTracker. This object will be continuouslypositioned to be right at the edge of the blade. Add an empty child tothe BladeEdgeTracker named SeparationPlane and move the game object sothat the X/Z plane is in line with the cutting edge of the blade (thisdefines where the mound and the terrain underneath it are divided).Note: The terrain modifier must be above separation plane or the controlsystem 22 will constantly dig and fill the area in between, creating abubbling effect. Assign the BladeEdgeTracker and the SeparationPlane tothe MoundHeightmap component. Add HeightmapSupportedAreas as children ofthe BladeEdgeTracker for the left and right side of the blade if needed(e.g. if the blade has side walls to help carry material). Add aHeightmapSupportedArea as a child of the MoundHeightmap object and setits bounding box to be just inside the bounds of the mound. Assign theMoundHeightmap into the “Terrains to ignore” list of thisHeightmapSupportedArea. This will prevent windrows from eroding into themound. Add an ErodeAndRaise component to the MoundHeightmap. Finally,apply to the prefab and assign the prefab to theMoundHeightmapController.

In some embodiments, the control system 22 includes simulation softwarefor Level I simulators for the Heavy Equipment Training Simulators whichwill train commercial users in the tasks associated with the operationand maneuverability of the equipment. The simulator system 10 willreplicate the functions and actions of a heavy equipment Machine Modelto train commercial users on the basic principles of operating theequipment to include locations and functions of controls andinstruments. The simulator system 10 is used to teach the basic skillsrequired to operate and maneuver the equipment, emphasizing efficientperformance and safety. The simulator system 10 will include thestandard commercial operator seat 28, controls for that heavy equipmentMachine Model 30, computer system 54, 42 inch LCD HD TV monitor(s)display(s) 38, and a head mounted display unit 20.

In some embodiments, the simulator system 10 contains the following keyfeatures:

a) Caterpillar Controls 34: Controls from actual machine are designedfor the simulator pod(s) and provide realistic interface forcontrolling, operating and maneuvering the machine. All controls sendPulse-Width Modulation (PWM), Button, or Controller Area Network (CAN)signals to a proprietary controller that interprets the machine signalsand converts them into CAN messages to transport over a small networkand back to the PC via a Universal Serial Bus through DirectInput. Thecontroller configuration allows for various dynamic controlconfigurations for multiple heavy equipment machines.

b) Operator Tasks: Operating techniques and procedures are practicedwith real-world activities as described in the training lessons.

c) Scoring: The score in each lesson is generated based on time,productivity, equipment damage (number of contacts with objects), andexercise success. The simulator system 10 tracks/reports numerousmetrics and generates student scores based on performance measures beingrecorded with the “final score” for a lesson being expressed in terms ofexercise success as well as reporting of performance measures andmetrics.

d) Safety: The control system 22 emphasizes safety while operating theequipment to increase the Student's knowledge level and awareness ofjobsite safety. The simulator system 10 tracks numerous safetyviolations during training lessons. Any time a safety violation isdetected, a message will be displayed indicating the type of violationand any other relevant information.

e) Real-time Feedback: The control system 22 provides real-time feedbackto the Student when mistakes are made and potential hazards areencountered.

f) Practice and Assessment Modes: The control system 22 operates in thefollowing modes: Practice Mode with help: Help functionality-Three iconswill be displayed on the left side of the screen. Two inactive and oneactive. Two inactive icons will be displayed in black & white and theActive will be displayed in full color. The inactive icons are 100×100and the active 120×120. After a predefined period of amount of time(this is set in SimU Campus™), without completion of objective graphicalhelp will be displayed at the lower, left of the screen (keeping text ata minimum). Procedure icons will be the official InternationalOrganization for Standardization™ (ISO) icons. The Help Icon will be theactual control image with arrows indicating proper direction of action.There will always be general instruction feed displaying across thebottom of the screen. These are step by step instructions to assist theuser with properly and safely using the machine to complete thenecessary exercise tasks. User will not receive a permanent score duringpractice mode, but the exercise metrics will be graded against the exammode criteria. Assessment Mode or Exam Mode: No Help functionality.There will be high level section instructions to assist with the flow ofthe exercise. These will be displayed across the bottom of the screen.User's score from each exercise ran will be retained within the SimUCampus™ Training Management software.

g) Realistic Virtual Environment: The system's realistic virtualenvironment 24 includes high fidelity visual system, depth cues andaudio cues associated with the actual equipment and other equipment andobjects in the jobsite.

h) Machine Performance: Modeling is based on the actual equipmentperformance data providing realistic response to the student andinteraction with the environment.

i) Languages. The software interface and performance statistics will beavailable in English as well as customer defined foreign languages.

j) Camera Views. The simulator system 10 allows the user to change theview by clicking the mouse for various views which include thefollowing: 1) In Cab View: User will have the ability to move the cameraaround as though they were moving their head in cab by using the mouse.Camera movements with the mouse will include: Position (X, Y). 2)Outside Cab View: User will have the ability to navigate the entire areaaround the machine through the orbit camera. The camera will integratethe mouse controls to assist with navigation. Mouse Wheel will allow theuser to zoom in/out. Moving the mouse will navigate the orbit camera infree space within the virtual environment. The orbit camera will limitthe user to stay above the terrain when navigating around with thecamera.

k) Training Manager. The simulator software must work with the SimUCampus™ Training Manager Software. The simulator software must accept aparameter(s) to tell what mission to start, what map to start, whetherto start in practice mode or assessment mode, and various otherparameters set through SimU Campus™ Training Manager Software.

Operator's Seat: The operator's seat 28 and the console has a variety ofadjustments in order to meet a wide range of Operator's needs.Adjustable Seat—The seat will have the ability to be adjusted by theuser for best positioning, and comfort during training. The seat canadjust in the following manner: Forward/Back; Recline Forward/Back;Headrest Raise/Lower. Safety Belt—The seat will be equipped with aSafety Belt, which will be connected to an electrical switch that willtoggle a signal in the software to indicate when the user has the beltfastened or not.

Training Exercises: 1) Walkaround; 2) Controls Familiarization; 3) OpenTraining; 4) Besides above standards, varies by machine model.

Exercise Passing: When a user operators the machine through the entireexercise and completes the necessary tasks the simulation will pass theuser. “Passing” in this situation does NOT mean the user has completedthe necessary tasks within the allotted ranges setup by the expert datawithin SimU Campus™ Training Records Management software. Once the usermakes it to the end of the exercise and shuts the machine down thefollowing should always happen. Once the simulation reaches thecompletion point (Key Off machine) all controls will be locked out. Notuser input through the controls will be recognized or cause any responsewithin the simulation. The simulation will display “Exercise Complete!”text at the bottom of the screen. The text will display in a greencolor. The screen will fade from color display to black and whitedisplay over the course of 15 seconds before completely shutting down.The simulation will write the exercise metrics and data to an XML filefor SimU Campus™ Training Records Management software to display withina report (shown in FIG. 26).

Exercise Failure: When a user operates the machine in an unsafe manner,fails to adhere to exercise instructions or various other incorrectactions, the simulation exercise could fail the user. In the event ofexercise failure the following should always happen. Once the simulationreaches a failure state all controls will be locked out. No user inputthrough controls will be recognized or cause any response within thesimulation. The simulation will display the failure reason at the bottomof the screen. Text will be displayed as, “FAILURE: [InstructionString]”or “FAILURE: [FailureReason]”: Where [InstructionString] is the currentinstruction; Where [FailureReason] is the explanation for cause offailure. The Text will display in a red color. Normal exercisecompletion routine will commence, but instead of “Exercise Complete!” itwill display the Failure information until the simulation closes. Thescreen will fade from color display to black and white display over thecourse of 15 seconds before completely shutting down. The simulationwill write the exercise metrics, data and failure information to an XMLfile for SimU Campus™ Training Records Management software to displaywithin a report.

Warning System: The warning system is designed to help inform the userof incorrect actions, unsafe practices, etc. so the user has a betterunderstanding of proper machine operation and what is consideredincorrect or unsafe. The warning system also helps the simulation keepthe user on-track with the necessary exercise steps and procedures. Ifmultiple warnings are present in the system, the most important problemis shown first (the warnings spreadsheet will have warning levels pleaserefer to it for more detail).

WarningTimer—parameter defined in the configuration file to set theamount of time between when an instruction is displayed until thesimulation starts warning the user for NOT completing the instruction,Default WarningTimer value is 20 seconds.

FailureTimer—parameter defined in the configuration file to set theamount of time between when an instruction is moved to a warning andwhen the simulation fails the user for NOT completing the instruction,Default FailureTimer value is 15 seconds.

When any exercise loads the warning system will be enabled and willdisplay warning information through the exercise if certain criteria arenot met. Some warnings will be displayed ONLY in practice mode.

Ignoring Instructions Warnings—these warnings are designed to inform theuser that they are not following proper procedures displayed on thescreen. If an instruction is displayed in practice mode, a timer isstarted once the instruction is displayed and the instruction willdisplay for the amount of time defined in the WarningTimer parameter. Ifthe instruction is NOT completed by the user after the definedWarningTimer period then the instruction will turn into a warning: Textwill be displayed as, “WARNING: [InstructionString]”, Where[InstructionString] is the current instruction. The Text will changecolors from Yellow to Red based on the elapsed time. Yellow text isdisplayed when the warning text is first displayed. Red text would bedisplayed when the system reaches the FailureTimer value with aninstructional warning. If the instruction is NOT completed by the userafter the defined FailureTimer period then the instruction will become afailure: Text will be displayed as, “FAILURE: [InstructionString]”,Where [InstructionString] is the current instruction. The Text willchange to a red color. Normal exercise completion routine will commence,but instead of “Exercise Complete!” it will display the Failureinformation until the simulation closes. This warning will increment an“Ignoring Instructions” metric each time it is generated.

Exercise Warnings—these warnings will occur when the user is performingactions outside of the intent of the exercise. These warnings include:Get Back On Course warning—If the user drives off course from thedesired path during an exercise they will receive this warning. Thiswarning will progress to exercise failure if NOT corrected. Return ToExercise Area warning—If the user drives away from the area required toperform the exercise they will receive this warning. This warning willprogress to exercise failure if NOT corrected. Fuel Level Low warning—Ifthe user allows the fuel level to drop less than 10% of the originalvolume they will receive this warning. This warning is a Level 1 warningand will also cause a warning to display on the Monitor system.Overspeed warning—If the user operates the machine in a fashion thatoverspeeds the engine they will receive this warning. This warning is aLevel 2 warning and will also cause a warning to display on the MonitorSystem AND the Alert indicator on the Monitor will blink until thesituation is corrected. This warning will progress to exercise failureif NOT corrected. Lift Overload warning—If the user attempts to liftmore than the machines defined/safe capacity they will receive thiswarning.

This warning is a Level 3 warning and will also cause a warning todisplay on the Monitor System AND the Alert indicator on the Monitorwill blink AND an audible alarm will sound. This warning will progressto exercise failure if NOT corrected. The warning text will display as,“WARNING: [WarningText]”, Where [WarningText] is the defined warninginformation in the Exercise Warnings.xls spreadsheet. The warning textwill change colors from yellow to red based on the elapsed time: Yellowtext is displayed when the warning is first displayed. Red text isdisplayed when the warning has progressed to a failure. Warnings that doNOT progress to failure will just flash in yellow text three times atthe bottom of the screen. This warning will increment an “IgnoringInstructions” metric each time one is generated.

Walkaround: This exercise will utilize the Caterpillar Machine Model andreview the proper pre-operation walkaround procedures. The user willstart at the base of the machine and work their way through thedifferent inspection points per section on the machine. The user willhave the ability to navigate between inspection sections usingdirectional arrows. The user will have the ability to either highlightthe proper inspection point or have the inspection point highlighted bypointing at the name in a list. At each inspection point a question willbe posed to the user relating to the part or inspection point. The userwill have the ability to indicate whether the inspection point is faultyor passes inspection. The user will work their way around the machinefrom base to cab in an organized manner every time the exercise is run.The exercise is complete when all inspection points have been reviewed.The number of faulty points will be defined in the SimU Campus™ Classsettings. These faulty points will be determined at random when theexercise is loaded. SimU Campus™ will write the value to the<WalkaroundFaults> configuration parameter when launching thesimulation.

The exercise will load and present the pre-exercise instructions/tipsscreen. The pre-exercise screen will contain an image of the machine inthe exercise environment performing the exercise. The pre-exercisescreen will contain pertinent exercise information/tips as bullet pointsnext to the image.

This is the first section of the walkaround: Each Section contains apre-defined number of inspection points. The user can navigate betweensections at-will but the entire exercise is NOT complete until they havemarked all inspection points. The section's walkaround points will belisted in the upper left corner of the user view. Each inspection pointwill have a transparent checkmark next to it: Once an inspection pointhas been reviewed the checkmark will change colors accordingly:Green=successfully indicated the part as good or faulty. When aninspection point is correctly marked the simulation will indicate with apassing sound as well. Red=Incorrectly indicated the part as good orfaulty. When an inspection point is incorrectly marked the simulationwill indicate with a failure sound as well. If the user hovers the mousepointer over an inspection point in the list: The text in the list willchange color to yellow to indicate a highlight. The correspondingpart/inspection point on the machine section will also highlight yellowto assist users who are unfamiliar with the part/inspection point. Ifthe user hovers the mouse pointer over an inspection point on themachine: The text in the list will change color to yellow to indicate ahighlight. The corresponding part/inspection point on the machinesection will also highlight yellow. Each Section will have navigationalarrows on either side of the user view to allow the user to move betweensections on the machine. The Left side arrow will move the user to thenext inspection section to the left of the current section. The Rightside arrow will move the user to the next inspection section to theright of the current section. The navigation will “fly” to certainpoints on the machine so the user has an up-close view of the inspectionpoints and parts. However, in the real world some of the inspectionpoints are visually inspected from the ground level or the Stick andBoom are lowered in such a manner that the user can be closer to theinspection points.

The walkaround will be divided into Ground Level, Engine/Platform Level,Cab Level inspections. Once all walkaround sections and inspectionpoints have been reviewed the exercise will end. Simulation will displaythe following message, “Exercise Complete!”. Text will display in greencolor. The simulation message, “Exercise Complete!” will not be affectedby the color changing. The simulation will compile the exercise resultsand output the metrics and data to an XML file in the appropriatelocation for the reporting tool to use.

Controls Familiarization: This exercise will utilize the CaterpillarMachine Model and review with the user the basic control operations ofthe machine. The control review will be randomized so that the order isnot always known by the user. During this exercise the simulation willdisplay an icon/control image helper system in the lower left corner ofthe user view. The user will have a set amount of time to perform therequested action by the simulation, and after that time they will failfor not following instructions.

Control Familiarization Helper System: The Control FamiliarizationHelper System is designed to assist the user becoming familiar with themachine. It will show an icon (from the machine's Caterpillar O&MMmanual) as well as the actual control in an image along with theon-screen simulation instructions. This helps give a visualrepresentation of the control to the user. The Helper System will beenabled for both practice and exam modes.

When the control familiarization exercise loads it will randomize theentire exercise and store the listing in memory. This will allow us todisplay future instructions (as icons) in a list for the user. For eachsection of the control familiarization the helper system will display atleast 3 instructions ahead of icons, if possible, in the lower leftcorner of the user view. When an instruction displays the icon will beactive: The icon coloring will change from Black/White/Grey to Yellow.The icon and instruction will display for 10 seconds allowing the userto properly perform the requested action. If an action is performed: Ifthe action performed is the correct action: The simulation will make acorrect action sound. The helper system will move (animate/flip) to thenext icon along with the simulation system's on-screen instruction text.If the action performed is NOT the correct action: The helper systemwill remain as is and the 10 second timer started above will continue tocount down. The simulation will note an incorrect action metric relatedto the control being used by the simulation system. If no action isperformed after 10 seconds: The icon will animate (flip) to the actualcontrol Image. The Control Image will increase in size from 100×100 to120×120. The Control Image and instruction will display for 10 secondsallowing the user to properly perform the requested action. If an actionis performed: If the action performed is the correct action: Thesimulation will make a correct action sound. The helper system will move(animate/flip) to the next icon along with the simulation system'son-screen instruction text. If the action performed is NOT the correctaction: The helper system will remain as is and the 10 second timerstarted above will continue to count down. The simulation will note anincorrect action metric related to the control being used by thesimulation system. If no action is performed after 10 seconds: The userwill fail the exercise for ignoring instructions.

The exercise will load and present the pre-exercise instructions/tipsscreen. The pre-exercise screen will contain an image of the machine inthe exercise environment performing the exercise. The pre-exercisescreen will contain pertinent exercise information/tips as bullet pointsnext to the image. The User will be prompted to honk the horn tocontinue past the screen once the environment has loaded.

Start-up Procedures: Once the world camera orients to the operator viewthe control reset form will be displayed. Please refer to the ControlReset section of this document for detail. After proper control resetthe user will be lead through the proper startup procedures. User willbe guided through the Startup Procedures. If at any time the operatorfails to perform a step, or if they perform a step incorrectly, theywill fail the exercise for incorrect machine startup. The simulationapplication will pass appropriate failure messages to the reportingtool.

After properly starting the machine the user will begin the exerciseprocedures: For this exercise the control listing is randomized duringthe initialization. So, the order of instructions will always bedifferent with the exception of the startup and shutdown sections. Theexercise will display control familiarization sections utilizing thehelper system defined above. The exercise will follow the same failuresystem as described above in the helper system.

After going through all controls on the machine (including switches,pedals and lockout) the user will be prompted to shutdown the machine.

Once the machine has been shut down and the key turned to the offposition the simulation will begin to shutdown: Simulation will displaythe following message, “Exercise Complete!”. Text will display in greencolor. The simulation will fade to Black and White. 15 seconds betweenpressing horn control and final Black and White display. The simulationmessage, “Exercise Complete!” will not be affected by the colorchanging. The simulation will compile the exercise results and outputthe metrics and data to an XML file in the appropriate My Documentslocation for the reporting tool to use.

Open Training: This exercise will utilize the Caterpillar Machine Modeland allow the user to review any skills in the construction environment.

The exercise will load and present the pre-exercise instructions/tipsscreen. The pre-exercise screen will contain an image of the machine inthe exercise environment performing the exercise. The pre-exercisescreen will contain pertinent exercise information/tips as bullet pointsnext to the image. The User will be prompted to honk the horn tocontinue past the screen once the environment has loaded.

Startup Procedures: Once the world camera orients to the operator viewthe control reset form will be displayed. Please refer to the ControlReset section of this document for detail. After proper control resetthe user will be lead through the proper startup procedures. User willbe guided through the Startup Procedures. The Startup Procedures will bethe same for EVERY exercise. If at any time the operator fails toperform a step, or if they perform a step incorrectly, they will failthe exercise for incorrect machine startup. The simulation applicationwill pass appropriate failure messages to the reporting tool.

After properly starting the machine the user will begin the exerciseprocedures: For Open Training there are no formal procedures or stepsother than starting the machine and practicing safe operatingprocedures. The Simulation will display the following text, “OpenTraining—Press ESCAPE To Exit When Finished”. The user does NOT have toproperly shut down the machine in this exercise. When the user exits theexercise the simulation will compile the exercise results and output themetrics and data to an XML file in the appropriate My Documents locationfor the reporting tool to use.

Referring to FIG. 28, in some embodiments, the control system 22 mayinclude a computer system 100 within which a set of instructions may beexecuted causing the machine to perform any one or more of the methods,processes, operations, or methodologies discussed herein. In an exampleembodiment, the machine operates as a standalone device or may beconnected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment. Themachine may be a server computer, a client computer, a personal computer(PC), or any machine capable of executing a set of instructions(sequential or otherwise) that specify actions to be taken by thatmachine. Further, while only a single machine is illustrated, the term“machine” shall also be taken to include any collection of machines thatindividually or jointly execute a set (or multiple sets) of instructionsto perform any one or more of the methodologies discussed herein.

The example computer system 100 includes a processor 56 (e.g., a centralprocessing unit (CPU) a graphics processing unit (GPU) or both), a mainmemory 58 and a static memory 60, which communicate with each other viaa bus 102. The computer system 100 may further include a video displayunit 104 (e.g., a liquid crystal display (LCD) or a cathode ray tube(CRT)). The computer system 100 also includes an alphanumeric inputdevice 106 (e.g., a keyboard), a cursor control device 108 (e.g., amouse), a drive unit 110, a signal generation device 112 (e.g., aspeaker) and a network interface device 114.

The drive unit 110 includes a machine-readable medium 116 on which isstored one or more sets of instructions (e.g., software 118) embodyingany one or more of the methodologies or functions described herein. Thesoftware 118 may also reside, completely or at least partially, withinthe main memory 58 and/or within the processor 56 during executionthereof by the computer system 100, the main memory 58 and the processor56 also constituting machine-readable media. The software 118 mayfurther be transmitted or received over a network 120 via the networkinterface device 114.

FIGS. 29-33 are flow charts of methods 200, 300, 400, 500, and 600illustrating algorithms that may be executed by the control system 22 tooperate the heavy equipment simulation system 10. The methods include aplurality of steps. Each method step may be performed independently of,or in combination with, other method steps. Portions of the methods maybe performed by any one of, or any combination of, the components of thecontrol system 22 and/or the heavy equipment simulation system 10.

Referring to FIG. 29, in method step 202, the processor 56 executes thesimulation software to initiate a training simulation using thesimulated heavy equipment vehicle 26 within the virtual environment 24upon receiving a user start-up input signal from the user via the heavyequipment vehicle control assembly 30.

In method step 204, the processor 56 initially operates in the non-VRmode when initiating the training simulation and only renders thevirtual environment on the display device assembly 18. The processor 56then initiates a start-up sequence that requires the user to perform asequence of control inputs using the heavy equipment vehicle controlassembly 30.

In method step 206, the processor 56 determines whether the user hascompleted the start-up sequence using the heavy equipment vehiclecontrol assembly 30. If the user has not completed the start-upsequence, the processor 56 may display a notification message on thedisplay device assembly 18 guiding the user through the remainingcontrol inputs required to complete the start-up sequence.

In method step 208, upon determining that the start-up sequence has beencompleted, the processor 56 enables a camera switching mode andestablishes the VR camera viewpoint location and the non-VR cameraviewpoint location. In one embodiment, the processor 56 is programmed toswitch operation from the non-VR mode to the VR mode upon determiningcompletion of the start-up sequence by the user and determining the VRheadset unit is being worn by the user.

In method steps 210 and 212, the processor 56 receives input from the VRheadset unit 20, and determines whether the VR headset unit 20 is beingworn by the user and is in position for operating the VR mode.

In method steps 214-220, upon determining the VR headset unit 20 isbeing worn by the user, the processor 56 renders the training simulationin the VR mode by enabling the VR camera viewpoint of the virtualenvironment 24, disabling the non-VR camera viewpoint, andsimultaneously rendering the VR camera viewpoint on the VR head mounteddisplay 42 and the display device assembly 18. The processor 56 may alsoswitch from operating in the VR mode to operating the non-VR mode upondetecting the VR headset unit 20 has been removed from the head of theuser.

In method steps 222-226, upon determining the VR headset unit 20 is notbeing worn by the user, the processor 56 renders the training simulationin the non-VR mode by enabling the non-VR camera viewpoint of thevirtual environment 24, disabling the VR camera viewpoint; and renderingthe non-VR camera viewpoint on the display device assembly 18.

Referring to FIG. 30, method 300 of training an operator is shown,according to some embodiments. An instruction module is optionallyinitiated 302 by an operator (i.e., student or user) or an instructor. Asimulation module 304 is then optionally initiated. One or moreinstruction modes are optionally selected 306. The operator interacts308 with one or more simulation modes of a heavy equipment or machinesimulator. The operator is evaluated using criteria substantiallysimilar to an original equipment manufacturer (OEM) criteria for anon-simulated heavy equipment model. The results are then optionallycompiled 310 and optionally transferred 312 to the instruction module.The results are then optionally output 314 as a visual display, printout or both.

Initiating a module can be performed by an instructor or operator. Thesimulator allows for an instructor to build a custom exercise program.The program can simulate equipment failure and incident response, forexample. In the instruction module, the instructor can select safetyprocedures, operational exercises and operational and equipmentparameters (e.g., engine speed). The instructor can be near thesimulator or interacting with the instruction module remotely, such asthrough a remote input device. Examples of remote input devices includesmart phone, tablet computer, and/or laptop computer.

Initiating the simulation module 304 can be done remotely or at thesimulator. The one or more instruction modes can be selected by eitherthe instructor or student. Instruction modes include at least a practicemode and exam/assessment mode or operational mode. The practice modeincludes help functionality not present in the assessment mode. Oneexample of the help mode includes a graphical control icon that appearson the display after a set time has elapsed without completing anexercise goal. The help mode may include instruction feed text on thedisplay, such as on the bottom of the display. The practice modeoptionally does not score a user, but can be graded against the exammode criteria.

If in exam mode or scored in practice mode, results are compiled 310either at the simulator or remotely (e.g., wirelessly transmitted to aremote device or computer). In the assessment mode, there may be sectioninstructions to assist with the flow of the exercise. These can bedisplayed across a portion of a display screen. The instruction moduleaccepts one or more parameters to indicate what exercise to start, whatmap to start, whether to start in practice mode or assessment mode, andvarious other parameters set through instruction module by an instructoror user.

The operator (i.e., student or user) interacts 308 with one or moresimulation modes, such as simulation exercises. Interacting 308 includesperforming one or more simulated exercises substantially similar to anOEM exercise for the non-simulated heavy equipment model. For example,the specific exercises that an OEM designates for training an operatoron a “real world” machine are substantially simulated within thesimulator. Further, the evaluation and criteria by which the operator isjudged in the simulation substantially or exactly mimics the criteriaused to evaluate “real world” operators of non-simulated machines orheavy equipment.

Examples of training exercises include walkaround, controlsfamiliarization, bucket placement, “raking the green”, “over the moon”,loading & off loading the machine from a trailer, trenching, truckloading, bench loading, setting trench box and pipe, backfilling,production cycle, quick coupler and open training.

Before simulating operating tasks, pre-operation tasks can be simulated.

Examples of pre-operation tasks include control reset, startupprocedures, safe machine operation, and shutdown procedures. Controlreset checks include making sure all controls are in the proper positionprior to starting the machine (e.g., on/off key position, hydraulic locklever position, fine swing control on/off, etc.). A horn signal may beused to indicate all controls are set properly. A user cannot pass thischeck if controls are not set properly (regardless of horn activation).

The controls familiarization exercise can utilize the machine and reviewthe basic control operations of the machine. The control review can berandomized so that the order is not always known by the user. Duringthis exercise the simulation can display an icon/control image helpersystem in the lower left corner of the user view. The user can have aset amount of time to perform the requested action by the simulation,and after that time they can fail for not following instructions.

Startup procedures can be prompted by on-screen text or after thecontrol reset exercise is completed. Examples of startup procedurechecks include fastening seat belt and warming up machine. Safe machineoperations exercise can include such actions carrying implements at asafe distance from the ground, avoiding terrain obstacles, fillingcontainers to appropriate levels, for example. Shutdown proceduresexercise requires a user to park on a level surface, reduce enginespeed, allow travel levers to return to correct position, and otheractions to ensure a safe shutdown and position the machine for asubsequently safe startup.

A warning system can be utilized and is designed to help inform the userof incorrect actions, unsafe practices, etc. so the user has a betterunderstanding of proper machine operation and what is consideredincorrect or unsafe. The warning system also helps the simulation keepthe user on-track with the necessary exercise steps and procedures. Ifmultiple warnings are present in the system, the most important problemcan be shown first.

Audible cues and sounds are implemented during an initial safetywalkaround, such as by correctly identifying a non-broken part,correctly identifying a broken part, incorrectly identifying a part asbroke, incorrectly identifying a part as non-broken, opening compartmentdoors, removing and replacing fluid level dipsticks, extracting andretracting the machine seatbelt.

During the simulation there can be an “environment” sound being played.This can provide normal background sounds to a typical constructionenvironment (wind, birds, machines/trucks driving, etc.). On top of the“environment” sound events occurring during the exercise, more soundscan be played. A few examples include operator entry, starting themachine, machine implement slams. Sounds can change during the exercise,such as when changing the engine speed control dial, putting load on theengine, moving the machine implements, etc.

An artificial intelligence (AI) component can be used to simulateon-site directions from a co-worker. The AI component is shown on theuser interface as a worker giving hand signals to the operator.

The Camera Helper System is designed to give the user a first personview of specific areas within the simulation that require another viewto get the full sense of depth. The Camera Helper System also shows howthe Quick Coupler control functions (as far as locking and unlocking thebucket), as well as giving the user a better view of the AI guide whenneeded. The placement of the Camera Helper can be set in code and cannotbe modified through a configuration setting. The placement of the CameraHelper View can be in the lower right corner of the simulation view.This allows easy viewing and keeps the users view of the environmentuninterrupted.

When in exercises where the user needs to load a truck, a truck helpercamera can be displayed so that the user can easily see where theirimplement is positioned relative to the truck bed.

During select exercises, a helper camera view can be enabled on one ormore of the displays. When the user spots a truck for loading (in boththe Truck Loading and Bench Loading exercises, for example) the helpercamera system can be enabled

When in exercises where the user needs to detach and attach machineimplements a helper camera can be displayed so that the user can easilysee where their coupler mechanism is located with respect to theimplement they are trying to attach. When detaching implements thecamera view can show the coupler joint with the implement and how themechanism unlocks when the user sends certain commands to the machine.

When in exercises where the user needs to be assisted by anotheroperator within the environment to help with object placement, thecamera can be displayed so that the user can easily see the signalingoperator (AI guide) as they guide the object safely to the finalplacement location. This camera view can be displayed in both practiceand exam mode to assist with the user viewing the AI guide's handsignals.

After interacting with simulation modules 308, the results are thentransferred 312 to the instruction module and output 314 onscreen orprinted. Results can be output 314 in real-time to instructor, studentor both. The results can be transferred 312 within the simulatorhardware or via a wireless or cellular signal, for example. Theinstruction module includes options for sorting and viewing the results.The simulator methods and system emphasizes safety while operating theequipment to increase the student's knowledge level and awareness ofjobsite safety. The simulator tracks numerous safety violations duringtraining lessons. Anytime a safety violation is detected, a message canbe displayed indicating the type of violation and any other relevantinformation. The system provides real-time feedback to the student whenmistakes are made and potential hazards are encountered.

When a user operates the machine through the entire exercise andcompletes the necessary tasks the simulation or instructor can pass theuser. The score in each lesson can be generated based on time,productivity, equipment damage (number of contacts with externalobjects), and exercise success. The simulator tracks/reports numerousmetrics and generates student scores based on performance measures beingrecorded with the “final score” for a lesson being expressed in terms ofexercise success as well as reporting of performance measures andmetrics.

Referring to FIG. 31, method 400 of simulating terrain is shown,according to some embodiment. A first simulated terrain can be contacted402 with a simulated component. The amount of first simulated terraincontacted is then calculated 404. Discrete shapes of the first simulatedterrain contacted are formed 406. The shapes can then be moved 408within a simulation to within proximity of a second simulated terrain.One or more triggering actions of the first simulated terrain can bedetected 410 in respect to the second simulated terrain. The secondsimulated terrain can be manipulated 412 to visually display an additionof the first simulated terrain contacted.

The first simulated terrain can be editable terrain, such as earth,debris, or other material that may interact with a simulated piece ofmachinery or equipment. The simulated component, such as an implement ofheavy equipment, contacts 402 the first simulated terrain. The implementcan be a bucket, blade or other attachment on a vehicle or heavyequipment. Contacting 402 can be digging, pushing, scraping, breaking,grabbing or other motion in which the simulated component and firstsimulated terrain interact.

Once the first simulated terrain is contacted 402, an amount of firstsimulated terrain is calculated 404. For example, if a simulated bucketdigs a portion of simulated earth from the first simulated terrain, acalculation is made as to the amount dug or interacted with. Thecalculation may be a volumetric calculation, for example. The firstsimulated terrain contacted can be formed into discrete shapes 406, suchas spheres or irregular shapes that visually represent the terrain beinginteracted with. The size and number of shapes formed 406 may affect theperformance and processing needs of the simulation.

Once the shapes are formed 406, such as dirt clods in a bucket, they canbe moved 408 within the simulation to near or within proximity to asecond simulated terrain. The second simulated terrain can be positionedin a layer underneath the visual layer of the desired or possiblereceiving locations (i.e., simulated receiving component) of the firstsimulated terrain contacted. For example, the second simulated terraincan be positioned underneath the bed of truck or trailer in which theoperator of the simulator is directed to place the material or firstsimulated terrain contacted. The simulated receiving component orreceiving location can be one or more of a truck bed, dumpster, trenchand trailer.

One or more triggering actions of the first simulated terrain can bedetected 410 independently or in respect to the second simulatedterrain. For example, the triggering action can be one or moreorientations of the first simulated terrain or the simulated componentinteracting with the terrain, independently or in respect to theposition of the second simulated terrain. The triggering action canindependently be a certain simulated velocity of the first simulatedterrain contacted or simulated component. Once the triggering action isdetected 410, the first terrain contacted interacts with the secondsimulated terrain. For example, at a certain velocity or orientation,the bucket can release the simulated earth into a simulated trailer bed.The second simulated terrain, positioned underneath or integrated withthe visual display of the trailer bed then responds or is manipulated412 to visually display the addition of such material into the bed. Thevolume or calculated first simulated terrain can be meshed with secondsimulated terrain and visually display local peaks, piles or generallyraise the height of the second simulated terrain to indicate theaddition of the material to that simulated location (i.e., the secondsimulated terrain). The spheres of first simulated terrain contacted areoptionally destroyed or smoothed out, according to visual preferencewithin the simulation.

Referring to FIG. 32, method 500 of simulating a physical interaction ina simulator is shown, according to some embodiments. Simulated physicalcontact zones between two or more simulated objects can be identified502. One or more interactions in the contact zones can be defined 504separately from the simulated environmental or equipment object rules.For example, when the environment (i.e., dirt or ground) and a piece ofequipment (e.g., bucket) interact, the physical rules for theirinteraction are defined similar to many types of collisions foundwritten into the software package. An example of physical contact zoneincludes regions in which objects in the environment are expected tocollide or interact. In order to smooth the visual look of theinteraction, reduce the resource drain to calculate the physics of theinteraction and overall optimize the interaction, the specific zone canbe isolated and defined (by rules separate from other interactions inthe environment).

In order for equipment interactions to display realistically withoutsignificantly affecting performance, virtual contact zones or joints canbe created between components. For example, once a component moves intoa physical contact zone, real time physics of movement is eliminated oraltered and a separate rules system is enacted, such as removing harshcollisions between objects. Such a system allows for smoother, morerealistic physical interactions between components on screen, withoutdragging performance.

Referring to FIG. 33, method 600 of training an operator of a machine isshown, according to some embodiments. A frame guide for an exercise canbe established or formed 602. Movement of one or more pieces ofequipment or vehicles can be simulated 604. The frame guide can bealtered 606 in response to the movement. The frame guide may be one ormore wire frame models showing the proper placement of implements,vehicles or portions of equipment during exercises. As the user movesthe equipment towards the proper position, the alteration of the frameguide may include change color, opacity, size, or other visual cue.

The frame guide can be utilized to give the user a visual sense of wheretheir current location of the implement is in relation to the wire framemodel they are trying to line up with. The system can calculate a valuebased on the position (x, y, z) and orientation (yaw, pitch, roll) ofthe current implement location compared to the predefined location theuser is trying to align with. The calculated value can then betranslated to a color shader, which can cause the wire frame model tochange colors that can help guide the user to proper implementalignment. Examples of features that can utilize the wire frame helperinclude bucket, stick, boom, tracks, trench box and pipe. The wire frameguide can indicate depth of field to provide the user with depthperception while guiding a simulated component. The depth of fieldindicator can be separate from other indicators.

Certain systems, apparatus, applications or processes are describedherein as including a number of modules or mechanisms. A module or amechanism may be a unit of distinct functionality that can provideinformation to, and receive information from, other modules.Accordingly, the described modules may be regarded as beingcommunicatively coupled. Modules may also initiate communication withinput or output devices, and can operate on a resource (e.g., acollection of information). The modules may be implemented as hardwarecircuitry, optical components, single or multi-processor circuits,memory circuits, software program modules and objects, firmware, andcombinations thereof, as appropriate for particular implementations ofvarious embodiments.

What is claimed is:
 1. A heavy equipment simulation system forsimulating an operation of a heavy equipment vehicle in a virtualenvironment, comprising: a support frame; an operator input controlassembly coupled to the support frame for receiving input from a user,the operator input control assembly including: a heavy equipment vehiclecontrol assembly for use in operating a simulated heavy equipmentvehicle in the virtual environment, the heavy equipment vehicle controlassembly configured to generate first input signals based on the user'sinteraction with the heavy equipment vehicle control assembly; a motionactuation system coupled to the support frame for adjusting anorientation of the support frame with respect to a ground surface; adisplay device assembly configured to display the virtual environmentincluding the simulated heavy equipment vehicle, wherein the displaydevice assembly includes a plurality of display monitors coupled to asupport assembly, the support assembly configured to support theplurality of display monitors from the ground surface, the displaydevice assembly being spaced a distance from the support frame to enablethe user to view the virtual environment being displayed on theplurality of display monitors; a virtual reality (VR) headset unitadapted to be worn by the user, the VR headset unit including a VR headmounted display configured to display the virtual environment includingthe simulated heavy equipment vehicle, and position sensors configuredto detect an orientation of the VR headset unit and generate secondinput signals based on the detected orientation of the VR headset unit;and a control system including a processor coupled to a memory device,the processor programmed to: generate the virtual environment includingthe simulated heavy equipment vehicle; operate the simulated heavyequipment vehicle within the virtual environment based on the firstinput signals received from the heavy equipment vehicle controlassembly; operate the motion actuation system to adjust the orientationof the support frame with respect to the ground surface based onmovements of the simulated heavy equipment vehicle within the virtualenvironment; render the virtual environment including the simulatedheavy equipment vehicle on the VR head mounted display; and render thevirtual environment including the simulated heavy equipment vehicle onthe display device assembly.
 2. A heavy equipment simulation system forsimulating an operation of a heavy equipment vehicle in a virtualenvironment, comprising: a support frame; an operator input controlassembly coupled to the support frame for receiving input from a user,the operator input control assembly including: a heavy equipment vehiclecontrol assembly for use in operating a simulated heavy equipmentvehicle in the virtual environment, the heavy equipment vehicle controlassembly configured to generate first input signals based on the user'sinteraction with the heavy equipment vehicle control assembly; a motionactuation system coupled to the support frame for adjusting anorientation of the support frame with respect to a ground surface; adisplay device assembly configured to display the virtual environmentincluding the simulated heavy equipment vehicle; a virtual reality (VR)headset unit adapted to be worn by the user, the VR headset unitincluding a VR head mounted display configured to display the virtualenvironment including the simulated heavy equipment vehicle, andposition sensors configured to detect an orientation of the VR headsetunit and generate second input signals based on the detected orientationof the VR headset unit; and a control system including a processorcoupled to a memory device, the processor programmed to: generate thevirtual environment including the simulated heavy equipment vehicle;operate the simulated heavy equipment vehicle within the virtualenvironment based on the first input signals received from the heavyequipment vehicle control assembly; operate the motion actuation systemto adjust the orientation of the support frame with respect to theground surface based on movements of the simulated heavy equipmentvehicle within the virtual environment; render the virtual environmentincluding the simulated heavy equipment vehicle on the VR head mounteddisplay; and render the virtual environment including the simulatedheavy equipment vehicle on the display device assembly, wherein theprocessor is programmed to operate in a VR display mode by: establishinga VR camera viewpoint location within a cab of the simulated heavyequipment vehicle within the virtual environment; rendering a VR cameraviewpoint of the virtual environment on the VR head mounted displaybased on the VR camera viewpoint location; monitoring an orientation ofthe VR headset unit based on the second input signals received from theVR headset unit; and adjusting an orientation of the rendered VR cameraviewpoint based on changes in the orientation of the VR headset unit. 3.The heavy equipment simulation system of claim 2, wherein the processoris further programmed to operate in the VR display mode bysimultaneously rendering the VR camera viewpoint of the virtualenvironment on the display device assembly.
 4. The heavy equipmentsimulation system of claim 3, wherein the VR headset unit includes aninfrared sensor configured to detect a position of the user's hands, theprocessor is further programmed to operate in the VR display mode byrendering images of simulated user hands within the virtual environmentbased on the detected position of the user's hands.
 5. The heavyequipment simulation system of claim 2, wherein the processor isprogrammed to operate in a non-VR display mode by: establishing a non-VRcamera viewpoint location within the cab of the simulated heavyequipment vehicle within the virtual environment; and rendering a non-VRcamera viewpoint of the virtual environment on the display deviceassembly based on the non-VR camera viewpoint location.
 6. The heavyequipment simulation system of claim 5, wherein the non-VR cameraviewpoint location is adjacent to the VR camera viewpoint locationwithin the cab of the simulated heavy equipment vehicle.
 7. The heavyequipment simulation system of claim 5, wherein the operator inputcontrol assembly includes a user input device configured to generatethird input signals based on the user's interaction with the user inputdevice for adjusting an orientation of a displayed viewpoint of thevirtual environment; and wherein the processor is further programmed tooperate in the non-VR display mode by adjusting an orientation of therendered non-VR camera viewpoint based on the third input signalsreceived from the user input device.
 8. The heavy equipment simulationsystem of claim 5, wherein the processor is further programmed to notrender the VR camera viewpoint on the VR head mounted display whenoperating in the non-VR camera mode.
 9. The heavy equipment simulationsystem of claim 5, wherein the processor is programmed to operate in theVR mode or the non-VR mode based on the detected orientation of the VRheadset unit.
 10. The heavy equipment simulation system of claim 9,wherein the VR headset unit is adapted to be worn on a head of the user,the processor is further programmed to switch from operating in the VRmode to operating the non-VR mode upon detecting the VR headset unitbeing removed from the head of the user.
 11. The heavy equipmentsimulation system of claim 10, wherein the processor is programmed to:initiate a training simulation using the simulated heavy equipmentvehicle within the virtual environment upon receiving a user start-upinput signal from the user via the heavy equipment vehicle controlassembly; and initiate a camera switching mode by: determining whetherthe VR headset unit is being worn by the user; and rendering thetraining simulation in the VR mode upon determining the VR headset unitis being worn by the user by: enabling the VR camera viewpoint of thevirtual environment; disabling the non-VR camera viewpoint; andsimultaneously rendering the VR camera viewpoint on the VR head mounteddisplay and the display device assembly; and render the trainingsimulation in the non-VR mode upon determining the VR headset unit isnot being worn by the user by: enabling the non-VR camera viewpoint ofthe virtual environment; disabling the VR camera viewpoint; andrendering the non-VR camera viewpoint on the display device assembly.12. The heavy equipment simulation system of claim 8, wherein theprocessor is further programmed to: initially operate in the non-VR modewhen initiating the training simulation; initiate a start-up sequencerequiring the user to perform a sequence of control inputs using theheavy equipment vehicle control assembly; and switch operation from thenon-VR mode to the VR mode upon determining completion of the start-upsequence by the user and determining the VR headset unit is being wornby the user.
 13. A method of operating a heavy equipment simulationsystem, the heavy equipment simulation system including a support frame,a heavy equipment vehicle control assembly for use in operating asimulated heavy equipment vehicle in the virtual environment andconfigured to generate first input signals based on the user'sinteraction with the heavy equipment vehicle control assembly, a motionactuation system coupled to the support frame for adjusting anorientation of the support frame with respect to a ground surface, adisplay device assembly configured to display the virtual environmentincluding the simulated heavy equipment vehicle, a VR headset unitadapted to be worn by the user and including a VR head mounted displayand position sensors configured to detect an orientation of the VRheadset unit and generate second input signals based on the detectedorientation of the VR headset unit, and a control system including aprocessor coupled to a memory device, the method comprising theprocessor performing the steps of: generating the virtual environmentincluding the simulated heavy equipment vehicle; operating the simulatedheavy equipment vehicle within the virtual environment based on thefirst input signals received from the heavy equipment vehicle controlassembly; operating the motion actuation system to adjust theorientation of the support frame with respect to the ground surfacebased on movements of the simulated heavy equipment vehicle within thevirtual environment; rendering the virtual environment including thesimulated heavy equipment vehicle on the VR head mounted display; andrendering the virtual environment including the simulated heavyequipment vehicle on the display device assembly, the processorperforming the steps of operating in a VR display mode by: establishinga VR camera viewpoint location within a cab of the simulated heavyequipment vehicle within the virtual environment; rendering a VR cameraviewpoint of the virtual environment on the VR head mounted displaybased on the VR camera viewpoint location; monitoring an orientation ofthe VR headset unit based on the second input signals received from theVR headset unit; and adjusting an orientation of the rendered VR cameraviewpoint based on changes in the orientation of the VR headset unit.14. The method of claim 13, including the processor performing the stepof operating in the VR display mode by simultaneously rendering the VRcamera viewpoint of the virtual environment on the display deviceassembly.
 15. The method of claim 13, including the processor performingthe step of operating in a non-VR display mode by: establishing a non-VRcamera viewpoint location within the cab of the simulated heavyequipment vehicle within the virtual environment; rendering a non-VRcamera viewpoint of the virtual environment on the display deviceassembly based on the non-VR camera viewpoint location; and disablingrendering on the VR head mounted display when operating in the non-VRcamera mode.
 16. The method of claim 15, including the processorperforming the steps of: initiating a training simulation using thesimulated heavy equipment vehicle within the virtual environment uponreceiving a user start-up input signal from the user via the heavyequipment vehicle control assembly; and initiating a camera switchingmode by: determining whether the VR headset unit is being worn by theuser; and rendering the training simulation in the VR mode upondetermining the VR headset unit is being worn by the user by: enablingthe VR camera viewpoint of the virtual environment; disabling the non-VRcamera viewpoint; and simultaneously rendering the VR camera viewpointon the VR head mounted display and the display device assembly; andrender the training simulation in the non-VR mode upon determining theVR headset unit is not being worn by the user by: enabling the non-VRcamera viewpoint of the virtual environment; disabling the VR cameraviewpoint; and rendering the non-VR camera viewpoint on the displaydevice assembly.
 17. The method of claim 16, including the processorperforming the steps of: initially operating in the non-VR mode wheninitiating the training simulation; initiating a start-up sequencerequiring the user to perform a sequence of control inputs using theheavy equipment vehicle control assembly; and switching operation fromthe non-VR mode to the VR mode upon determining completion of thestart-up sequence by the user and determining the VR headset unit isbeing worn by the user.
 18. One or more non-transitory computer-readablestorage media, having computer-executable instructions embodied thereon,wherein when executed by at least one processor, the computer-executableinstructions cause the at least one processor to: generate a virtualenvironment including the simulated heavy equipment vehicle; operate thesimulated heavy equipment vehicle within the virtual environment basedon the first input signals received from a heavy equipment vehiclecontrol assembly; operate a motion actuation system to adjust theorientation of a support frame with respect to a ground surface based onmovements of the simulated heavy equipment vehicle within the virtualenvironment; render the virtual environment including the simulatedheavy equipment vehicle on a VR head mounted display; and render thevirtual environment including the simulated heavy equipment vehicle on adisplay device assembly, the computer-executable instructions cause theat least one processor to operate in a VR display mode by: establishinga VR camera viewpoint location within a cab of the simulated heavyequipment vehicle within the virtual environment; rendering a VR cameraviewpoint of the virtual environment on the VR head mounted displaybased on the VR camera viewpoint location; monitoring an orientation ofthe VR headset unit based on the second input signals received from theVR headset unit; and adjusting an orientation of the rendered VR cameraviewpoint based on changes in the orientation of the VR headset unit.19. A method of operating a heavy equipment simulation system, the heavyequipment simulation system including a support frame, a heavy equipmentvehicle control assembly for use in operating a simulated heavyequipment vehicle in the virtual environment and configured to generatefirst input signals based on the user's interaction with the heavyequipment vehicle control assembly, a motion actuation system coupled tothe support frame for adjusting an orientation of the support frame withrespect to a ground surface, a display device assembly configured todisplay the virtual environment including the simulated heavy equipmentvehicle, wherein the display device assembly includes a plurality ofdisplay monitors coupled to a support assembly, the support assemblyconfigured to support the plurality of display monitors from the groundsurface, the display device assembly being spaced a distance from thesupport frame to enable the user to view the virtual environment beingdisplayed on the plurality of display monitors, a VR headset unitadapted to be worn by the user and including a VR head mounted displayand position sensors configured to detect an orientation of the VRheadset unit and generate second input signals based on the detectedorientation of the VR headset unit, and a control system including aprocessor coupled to a memory device, the method comprising theprocessor performing the steps of: generating the virtual environmentincluding the simulated heavy equipment vehicle; operating the simulatedheavy equipment vehicle within the virtual environment based on thefirst input signals received from the heavy equipment vehicle controlassembly; operating the motion actuation system to adjust theorientation of the support frame with respect to the ground surfacebased on movements of the simulated heavy equipment vehicle within thevirtual environment; rendering the virtual environment including thesimulated heavy equipment vehicle on the VR head mounted display; andrendering the virtual environment including the simulated heavyequipment vehicle on the plurality of display monitors of the displaydevice assembly.
 20. One or more non-transitory computer-readablestorage media, having computer-executable instructions embodied thereon,wherein when executed by at least one processor, the computer-executableinstructions cause the at least one processor to: generate a virtualenvironment including the simulated heavy equipment vehicle; operate thesimulated heavy equipment vehicle within the virtual environment basedon the first input signals received from a heavy equipment vehiclecontrol assembly; operate a motion actuation system to adjust theorientation of a support frame with respect to a ground surface based onmovements of the simulated heavy equipment vehicle within the virtualenvironment; render the virtual environment including the simulatedheavy equipment vehicle on a VR head mounted display; and render thevirtual environment including the simulated heavy equipment vehicle on adisplay device assembly, wherein the display device assembly includes aplurality of display monitors coupled to a support assembly, the supportassembly configured to support the plurality of display monitors from aground surface, the display device assembly being spaced a distance fromthe support frame to enable the user to view the virtual environmentbeing displayed on the plurality of display monitors.